Skip to content

Comments

AIP-67 - Multi-team: AwsBatchExecutor per team executor config#60920

Merged
ferruzzi merged 2 commits intoapache:mainfrom
anishgirianish:feature/aip-67-per-team-executer-config
Jan 26, 2026
Merged

AIP-67 - Multi-team: AwsBatchExecutor per team executor config#60920
ferruzzi merged 2 commits intoapache:mainfrom
anishgirianish:feature/aip-67-per-team-executer-config

Conversation

@anishgirianish
Copy link
Contributor

@anishgirianish anishgirianish commented Jan 22, 2026 •
edited
Loading

Summary

Add multi-team support to the AWS Batch Executor,

Changes

  • Add supports_multi_team: bool = True flag to AwsBatchExecutor
  • Move class-level MAX_SUBMIT_JOB_ATTEMPTS to instance-level max_submit_job_attempts
  • Update all conf.get() calls to use self.conf.get() for team-specific configuration
  • Change _load_submit_kwargs() from static method to instance method
  • Update build_submit_kwargs(conf) to take conf as required parameter
  • Add test_team_config test to verify team isolation works
  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@boring-cyborg boring-cyborg bot added area:providers provider:amazon AWS/Amazon - related issues labels Jan 22, 2026
@anishgirianish anishgirianish force-pushed the feature/aip-67-per-team-executer-config branch from d8e705d to 538035c Compare January 22, 2026 04:54
@anishgirianish anishgirianish mentioned this pull request Jan 22, 2026
Open
8 tasks
@o-nikolas
Copy link
Contributor

o-nikolas commented Jan 22, 2026

Thanks for the PR, I will review it in the next day or so!

o-nikolas
o-nikolas reviewed Jan 23, 2026
View reviewed changes
Copy link
Contributor

@o-nikolas o-nikolas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code looks good. Did you do any manual testing? If so can you describe the testing you did and the results you saw?

@anishgirianish
Copy link
Contributor Author

anishgirianish commented Jan 24, 2026 •
edited
Loading

Hi @o-nikolas, thanks for the review!

Yes, I did some manual testing. Set up AWS Batch (Fargate) to verify the changes work end-to-end.

Built a Docker image with the PR changes and ran it on AWS Batch.

Job succeeded:

aws-batch-console-job-succeeded

Logs output:

cloud-watch-logs-output

The logs show:
AwsBatchExecutor imported successfully
supports_multi_team: True

Image in ECR:

ecr-repository-image-upload

Everything works - the executor loads correctly with supports_multi_team enabled, and the config changes (moving to
self.conf and instance-level max_submit_job_attempts) don't break anything.

The test_team_config unit test covers the team isolation logic.

Let me know if you'd like me to test anything else!

Thank you.

o-nikolas
o-nikolas approved these changes Jan 26, 2026
View reviewed changes
Copy link
Contributor

@o-nikolas o-nikolas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Thanks for testing and contribution!

vincbeck
vincbeck approved these changes Jan 26, 2026
View reviewed changes
Copy link
Contributor

@vincbeck vincbeck left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

ferruzzi
ferruzzi approved these changes Jan 26, 2026
View reviewed changes
Copy link
Contributor

@ferruzzi ferruzzi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks reasonable.

@ferruzzi ferruzzi merged commit 75b4a29 into apache:main Jan 26, 2026
89 checks passed
@vincbeck vincbeck mentioned this pull request Jan 28, 2026
Closed
85 tasks
shreyas-dev pushed a commit to shreyas-dev/airflow that referenced this pull request Jan 29, 2026
@anishgirianish @shreyas-dev
AIP-67 - Multi-team: AwsBatchExecutor per team executor config (apach…
7aab7b0 
…e#60920)
shashbha14 pushed a commit to shashbha14/airflow that referenced this pull request Feb 2, 2026
@anishgirianish
AIP-67 - Multi-team: AwsBatchExecutor per team executor config (apach…
7c40f3a 
…e#60920)
jason810496 pushed a commit to abhijeets25012-tech/airflow that referenced this pull request Feb 3, 2026
@anishgirianish @jason810496
AIP-67 - Multi-team: AwsBatchExecutor per team executor config (apach…
47aa9dc 
…e#60920)
jhgoebbert pushed a commit to jhgoebbert/airflow_Owen-CH-Leung that referenced this pull request Feb 8, 2026
@anishgirianish @jhgoebbert
AIP-67 - Multi-team: AwsBatchExecutor per team executor config (apach…
0f5dfaa 
…e#60920)
choo121600 pushed a commit to choo121600/airflow that referenced this pull request Feb 22, 2026
@anishgirianish @choo121600
AIP-67 - Multi-team: AwsBatchExecutor per team executor config (apach…
8afdd42 
…e#60920)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ferruzzi ferruzzi ferruzzi approved these changes

@o-nikolas o-nikolas o-nikolas approved these changes

@vincbeck vincbeck vincbeck approved these changes

Assignees

No one assigned

Labels

area:providers provider:amazon AWS/Amazon - related issues

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@anishgirianish @o-nikolas @ferruzzi @vincbeck